Distributed systems can involve many nodes. In fact, these nodes can number in the tens, hundreds, thousands, millions or more nodal instances. Each instance may be, for example, a process, an application, a physical device, some combination thereof, and so forth. The individual nodes of a distributed system can operate interactively with one other, with two other, or with many other nodes of the distributed system. Such interactions may occur once or be repeated.
The many nodes of a distributed system usually communicate messages between and amongst each other. Each node also functions locally by acting on local resources. These various actions and interactions result in many different non-deterministic concurrences happening within the distributed system. As a result of these concurrent events and the sheer number of nodal instances, bugs in distributed systems are usually difficult to analyze. It is even more difficult to diagnose and/or identify the cause or causes of bugs in distributed systems.